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1.  Introduction 


In  June  2015  the  US  Army  Training  and  Doctrine  Command  published  US  Army 
Warfighter  Challenges }  The  purpose  of  this  document  was  to  identify  “enduring 
first-order  problems,  the  solutions  to  which  improve  the  combat  effectiveness  of 
the  current  and  future  force”.1  The  number  one  challenge  identified  was  “how  to 
develop  and  sustain  a  high  degree  of  situational  understanding  while  operating  in 
complex  environments  against  determined,  adaptive  enemy  organizations”.1  In 
military  command  and  control  operations,  situational  awareness  (SA)  is  crucial  to 
good  decision-making.  Defined  in  Army  Field  Manual  5-0,  situational 
understanding  is  the  product  of  applying  analysis  and  judgment  to  “relevant 
information”  to  determine  the  relationships  among  the  operational  and  mission 
variables  to  facilitate  decision-making.2 

One  of  the  central  challenges  associated  with  developing  good  situational 
understanding  originates  with  the  synthesis  of  timely  and  accurate  information  from 
the  vast  amounts  of  gathered  intelligence.  A  critical  aspect  of  that  synthesis  is  the 
determination  of  the  relative  importance  of  the  information  gathered,  with  that 
decision  being  a  function  of  operational  tempo  as  well  as  the  reliability,  content, 
and  latency  of  the  information.  For  the  military  commander  and  his/her  staff, 
separating  the  important  information  from  the  routine  has  become  a  labor-intensive 
task  and  impetus  for  this  research. 

Toward  that  end,  a  fuzzy-logic-based  model  was  developed  that  codifies  how 
intelligence  analysts  perceive  the  value  of  information  (Vol)  under  varying 
operational  tempos,  source  reliability,  information  content,  and  latency.3  The  Vol 
metric  is  built  using  a  fuzzy  associative  memory  approach  and  is  intended  to  codify 
the  subjective  deliberations  of  subject  matter  experts  (SMEs)  into  fuzzy  association 
rules  capable  of  approximating  the  SME  process  for  determining  Vol.4,5  Hanratty 
et  al.  detail  the  approaches  for  elicitation  of  expertise  from  intelligence  analysts  at 
the  US  Army  Research  Laboratory  (ARL)  and  for  the  construction  of  fuzzy 
association  rules  in  the  prototype  Vol  metric.6 

This  report  evaluates  the  impact  of  the  Vol  metric  within  a  constructive  simulation 
and  human  performance  model  known  as  Command,  Control,  and  Communications 
-  Techniques  for  Reliable  Assessment  of  Concept  Execution  (C3TRACE). 
C3TRACE  was  developed  by  the  ARL’s  Human  Research  and  Engineering 
Directorate  to  specifically  study  “information-driven”  decision-making  within 
command  and  control  structures.6 
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The  remainder  of  this  report  is  organized  as  follows:  The  details  of  the  experimental 
design,  data,  and  model  are  given,  followed  by  a  section  that  describes  the 
simulated  results.  Conclusions  and  future  work  are  presented  in  the  final  section. 

2.  Experimental  Design 

The  goal  of  this  experiment  is  to  determine  if  applying  a  value  rating  to  human 
intelligence  (HUMINT)  reports  and  then  analyzing  reports  in  order  of  highest  value 
first  can  increase  the  SA  gained  in  a  finite-time  period.  The  first  step  in  designing 
the  experiment  was  to  gain  a  deeper  understanding  of  the  HUMINT  analysis 
process  by  conducting  several  interviews  with  an  Army  officer  and  intelligence 
analyst.  These  interviews  were  the  basis  of  the  basic  model  and  parameters  used  to 
control  the  simulation.  The  researchers  developed  synthetic  data  sets  for  the 
simulation  and  executed  several  trials  with  the  simulation  to  determine  the  effect  of 
using  the  Vol  metric. 

2.1  Data 

The  data  for  this  experiment  consisted  of  simulated  HUMINT  reports.  The 
simulated  reports  consisted  of  a  Vol  rating  and  a  content  rating.  The  Vol  rating  was 
an  integer  value  in  the  range  10  to  0,  where  10  is  the  most  valuable.  During  the 
simulation,  the  Vol  rating  is  used  to  control  the  order  in  which  reports  are 
consumed.  The  content  rating  is  a  nominal  value  that  places  the  report  in  one  of  5 
categories:  Very  Good,  Good,  Average,  Bad,  and  Very  Bad.  The  content  rating 
determines  several  factors  during  the  simulation,  specifically  the  length  of  time  to 
analyze  the  report,  the  amount  of  SA  gained  from  this  analysis,  and  whether  or  not 
the  analysis  precipitates  further  research. 

Two  data  sets  were  generated  for  this  experiment.  Each  consisted  of  1 10  simulated 
reports  (Vol  and  content  rating)  in  a  comma-separated  value  file.  In  this  file  the 
nominal  content  rating  values  were  mapped  to  consecutive  increasing  integers, 
starting  with  Very  Good  =  1.  The  first  data  set  used  a  uniform  distribution  of  Vol 
values  (Table  1).  In  this  data  set  there  are  10  reports  for  each  Vol  value. 
Furthermore,  the  content  rating  of  each  report  is  assigned  in  a  method  that  creates 
a  linear  relationship  between  Vol  rating  and  SA  gained  from  analyzing  the  report. 
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Table  1  Uniform  data  set 


Vol 

Very  Good 

Good 

Average 

Bad 

Very  Bad 

10 

10 

0 

0 

0 

0 

9 

6 

4 

0 

0 

0 

8 

2 

8 

0 

0 

0 

7 

0 

8 

2 

0 

0 

6 

0 

4 

6 

0 

0 

5 

0 

0 

10 

0 

0 

4 

0 

0 

6 

4 

0 

3 

0 

0 

2 

8 

0 

2 

0 

0 

0 

8 

2 

1 

0 

0 

0 

4 

6 

0 

0 

0 

0 

0 

10 

In  this  linear  relationship,  the  SA  gained  by  analyzing  a  report  decreases  with  each 
decrement  in  Vol  value  (Fig  1).  To  calculate  this  relationship,  the  content  rating 
values  were  each  assigned  an  SA  value  (Table  2).  In  this  simulation  it  is  assumed 
that  reading  a  report  with  a  Vol  value  of  10  gains  the  analyst  the  maximum  SA. 
Thus,  all  10  reports  with  a  Vol  value  of  10  were  given  a  content  rating  of  Very 
Good.  Using  the  assigned  SA  values,  the  average  SA  gained  from  reading  a  report 
with  Vol  value  of  10  was  used  as  the  starting  point  for  generating  the  linear 
relationship.  A  decrease  of  10%  was  chosen  for  each  decrement  in  Vol  value. 
Finally,  a  combination  of  content  ratings  that  produced  the  desired  average  SA 
gained  was  selected  for  each  Vol  value. 


Uniform  Dataset:  Situational  Awareness  Gained 

versus  Vol  Rating 


Fig.  1  Relationship  between  Vol  and  SA  in  the  uniform  data  set 
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Table  2  Situational  awareness  gained  based  on  content  rating  of  report  analyzed 


Content 

rating 

SA 

Further 
investigation 
additional  SA 

Time  factor 

(s) 

Chance  to  investigate 
further 

(%> 

Very  Good 

4 

2 

150 

50 

Good 

3 

2 

150 

40 

Average 

2 

2 

150 

25 

Bad 

1 

2 

150 

15 

Very  Bad 

0 

2 

0 

10 

The  second  data  set  used  a  normal  distribution  of  Vol  values  (Table  3).  In  this  data 
set  the  amount  of  reports  for  each  Vol  value  was  determined  by  a  normal 
distribution  with  a  mean  of  5  and  encompassing  the  entire  Vol  scale  within  3 
standard  deviations  (Fig.  2).  In  addition,  the  content  rating  of  each  report  is  assigned 
in  the  previous  linear  relationship  method,  although  it  was  not  possible  to  achieve 
an  exact  fit  due  to  the  constraint  of  the  SA  values  (Fig.  3). 


Table  3  Normal  data  set 


Vol 

Very  Good 

Good 

Average 

Bad 

Very  Bad 

10 

1 

0 

0 

0 

0 

9 

1 

0 

0 

0 

0 

8 

1 

4 

0 

0 

0 

7 

0 

10 

3 

0 

0 

6 

0 

9 

13 

0 

0 

5 

0 

0 

26 

0 

0 

4 

0 

0 

13 

9 

0 

3 

0 

0 

3 

10 

0 

2 

0 

0 

0 

4 

1 

1 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

1 

Approved  for  public  release;  distribution  is  unlimited. 


4 


Normal  Distribution  Value  of  Information  Dataset 


Fig.  2  Normal  distribution  used  for  Vol  data  set 


Normal  Dataset:  Situational  Awareness  Gained  per 

Vol  Rating 


Fig.  3  Relationship  between  Vol  and  SA  in  the  normal  data  set 


The  reason  for  the  2  data  sets  is  to  examine  the  effect  of  the  data  landscape  on  the 
utility  of  the  Vol  metric.  The  uniform  distribution  data  set  will  allow  study  of  the 
effect  of  the  Vol  metric  on  the  SA  gained  from  analyzing  reports  when  the  quality 
(content  rating)  of  the  reports  is  spread  uniformly  from  Very  Good  to  Very  Bad.  In 
contrast,  the  normal  distribution  data  will  allow  a  study  of  these  effects  when  the 
majority  of  reports  consist  of  a  narrow  range  of  Vol  values. 
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2.2  Model 


The  model  used  in  this  experiment  is  created  and  exercised  using  the  C3TRACE 
software  (Fig.  4).  As  stated  previously,  the  model  is  the  product  of  several 
interviews  with  a  US  Army  officer  with  war  zone  HUMINT  analysis  experience. 
The  purpose  of  the  model  is  to  simulate  the  basic  steps  of  HUMINT  analysis.  Using 
anecdotal  knowledge  derived  from  the  SME  it  is  possible  to  assign  parameters  to 
the  steps  to  control  the  consumption  of  HUMINT  reports  and  the  SA  gained.  The 
overall  scenario  simulates  a  12-h  day  of  a  HUMINT  analysis.  The  analyst  is  tasked 
to  review  new  HUMINT  reports  and  present  mission-relevant  findings  to  the 
commander  by  the  end  of  the  day.  The  preparation  and  delivery  of  the  presentation 
requires  2  h,  thus  the  analyst  only  has  10  h  to  research  the  reports.  The  model 
combined  with  the  simulated  report  data  creates  the  experimental  system.  The 
experiment  will  answer  if  the  Vol  metric  can  increase  the  SA  (the  product  of 
researching  a  report  in  this  model)  gained  by  the  analyst  during  the  10-h  period  of 
time  in  this  simulation.  The  C3TRACE  reporting  features  are  used  to  record  the 
amount  of  reports  analyzed,  further  investigations  required,  and  the  total  amount  of 
situational  awareness  gained. 


Process TOPO  (1 6)  < 


{  Receipt of Mission  (1 1) 


»fiocess_HUMINT  (1_2)~<g)-< 


? - »^~innish_HUMINT  (1_4)  <^-L  Create_Report  (1_5) 


10 - 2 - 

^»^lnvestigate_Further (1_3) 


[  Process_SIGACT  (1_7)  < 


Fig.  4  C3TRACE  workflow 

The  model  is  not  intended  to  be  an  exhaustive  representation  of  the  complex  task 
of  HUMINT  analysis.  It  is  a  basic  process  enhanced  with  anecdotal  parameters. 
The  purpose  was  to  create  a  simulation  able  to  deliver  a  quantitative  first  look  at 
the  impact  of  the  Vol  metric  on  SA  gained.  If  the  results  precipitate  additional 
research,  the  complexity  of  the  model  may  be  revisited. 

2.2.1  Timekeeping 

During  the  simulation  C3TRACE  increments  the  elapsed  time  as  each  report  is 
analyzed.  This  is  an  important  task  because  the  scenario  allows  for  only  10  h  of 
analysis  time.  Multiple  timing  factors  were  used  in  the  following  formula  to 
calculate  the  time  elapsed  as  a  result  of  analyzing  a  report: 
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Report  Analysis  Time  =  Analysis  Baseline  Time  Factor 
+  Report  Time  Factor 

+  Further  Investigation  Time  Factor  *  Number  of  Further 
Investigations 

+Time  between  Reports 

Several  of  the  factors  are  constants  in  this  experiment,  derived  from  the  anecdotal 
experience  of  the  SME.  The  Analysis  Baseline  Time  Factor  is  always  60  s,  the 
Further  Investigation  Time  Factor  is  always  600  s,  and  the  Time  between  Reports 
is  60  s.  The  Report  Time  Factor  and  Number  of  Further  Investigations  are 
determined  by  the  simulated  content  rating  of  the  report  (Table  2). 

2.2.2  Situational  Awareness  Gained 

Similar  to  timekeeping,  the  SA  gained  in  the  simulation  is  tracked  by  C3TRACE. 
Each  time  a  report  is  completely  processed  or  a  further  investigation  occurs,  a 
constant  value  (Table  2)  is  added  to  the  total  SA  gained  in  the  simulation.  In  this 
model,  further  investigation  is  considered  a  deeper  exploration  of  the  information 
in  the  original  report,  requiring  additional  research  and  analysis.  This  is  a  time- 
consuming  process  as  the  Further  Investigation  Time  Factor  demonstrates. 
Furthermore,  the  result  of  the  further  investigation  may  vary,  but  in  this  simulation 
it  is  set  equivalent  to  the  SA  gained  from  analyzing  an  average  report.  The  SA 
gained  will  be  used  to  evaluate  the  performance  of  a  given  simulation. 

2.2.3  C3TRACE  Functions 

The  model  is  constructed  of  functions  (Fig.  4)  that  represent  basic  steps  in  the 
analysis  of  a  HUMINT  report.  Each  function  may  increment  the  elapsed  time  and/or 
the  SA  gained  given  the  content  rating  of  report  being  processed.  The  functions  are 
described  in  more  detail  in  the  following. 

Receipt_of_Mission  is  the  initial  function  in  the  model.  It  serves  to  maintain  a 
FIFO  (first  in,  first  out)  queue  of  HUMINT  reports.  During  interviews  it  was 
discovered  that  the  analyst  preferred  to  completely  process  a  report  before  moving 
on  to  the  next  report.  Thus,  this  function  will  not  begin  processing  a  new  report 
until  the  current  report  has  processed  all  the  way  through  the  model.  Furthermore, 
new  reports  will  only  begin  processing  if  the  simulation  time  is  below  the  deadline 
threshold,  which  in  our  experiment  is  10  h.  If  a  report  meets  these  requirements  it 
is  passed  to  the  Process_HUMINT  function. 
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Process_HUMINT  is  the  function  in  the  model  that  simulates  analyzing  a  report. 
In  this  function  the  SA  is  incremented,  and  the  time  spent  analyzing  the  report  is 
calculated  (time-elapsed  formula).  When  this  function  has  completed  its  task,  the 
model  has  the  option  of  2  paths.  It  can  either  advance  to  Investigate  Further  or 
FinishHUMINT.  The  chance  to  Investigate  Further  is  given  in  Table  2  and  is 
dependent  on  the  content  rating  of  the  report.  If  the  report  does  not  advance  to 
Investigate_Further,  it  goes  to  Finish_HUMINT  by  default.  C3TRACE  uses  a 
random  seed  on  each  run  of  the  model  for  use  in  all  chance-based  calculations. 
Hence,  the  path  taken  will  be  variable  between  runs. 

Investigate_Further  is  the  function  in  the  model  that  simulates  an  analyst 
performing  additional  research  into  a  report.  In  this  function  the  SA  is  incremented 
and  the  time  spent  analyzing  the  report  is  calculated.  Each  execution  of  this  function 
increments  the  elapsed  time  and  the  SA  gained  by  constant  values.  They  are  set  as 
constant  in  this  function  to  avoid  adding  recursive  complexity  by  simulating  in 
detail  the  information  discovered  during  further  investigation.  When  this  function 
has  completed  its  task,  the  model  has  the  option  of  2  paths,  just  like  the 
Process_HUMINT  function.  It  can  either  return  to  Investigate_Further  or  advance 
to  Finish  HUMINT.  The  chance  to  Investigate_Further  is  again  given  in  Table  2 
and  is  dependent  on  the  content  rating  of  the  report.  If  the  report  does  not  advance 
to  Investigate  Further,  it  goes  to  Finish_HUMINT  by  default.  The  recursive 
execution  of  Investigate  Further  may  only  occur  3  times  before  the  function  is 
forced  to  take  the  default  path  to  Finish  HUMINT. 

Finish_HUMINT  is  the  function  where  a  report  is  considered  processed.  If  the 
total  time  elapsed  in  the  simulation  is  over  the  10-h  deadline,  the  model  continues 
to  Create_Report.  Otherwise,  Receipt  of  Mission  is  unlocked  to  begin  processing 
the  next  report  in  the  queue. 

Create_Report  is  the  last  function  in  the  model.  When  the  elapsed  time  processing 
reports  is  greater  than  10  h,  this  function  is  executed.  The  function  increments  the 
elapsed  time  by  2  h  to  simulate  the  time  spent  preparing  and  presenting  the  analysis 
to  the  commander.  Thus,  the  total  time  spent  in  the  model  is  12  h.  After  the  function 
terminates,  the  total  SA  gained,  reports  processed,  and  further  investigations 
performed  are  recorded. 
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2.3  Experiment 


The  goal  of  the  simulation  is  to  record  the  value  of  the  SA  gained  from  analyzing 
as  many  reports  as  possible  within  a  10-h  time  limit  of  the  simulation.  The  order 
that  the  reports  are  analyzed  is  determined  by  the  Vol  value.  At  the  beginning  of  a 
trial,  the  report  file  is  ingested  by  C3TRACE,  and  the  order  of  the  individual  reports 
is  randomized  by  performing  a  Fisher-Yates  Shuffle.  Then  a  portion  of  the  reports 
from  the  beginning  of  the  shuffled  list  is  sorted  by  Vol  value  to  place  the  highest- 
value  reports  in  a  position  to  be  analyzed  first.  The  portion  of  the  reports  that  was 
sorted  is  called  the  “queue”,  and  its  size  was  changed  between  trials.  After  the 
sorted  reports  in  the  queue  are  analyzed,  the  remaining  reports  are  analyzed  in  the 
random  order  in  which  they  were  shuffled  at  the  beginning  of  the  simulation.  For 
each  data  set  there  were  1 1  trials.  The  first  trial  had  a  queue  of  zero,  meaning  no 
sorting  of  the  reports  was  performed.  This  is  the  worst-case  scenario,  as  the  reports 
are  ingested  in  a  random  manner  according  to  the  initial  shuffle.  At  the  beginning 
of  each  successive  trial,  the  queue  size  is  increased  by  10,  culminating  in  the  1 1th 
trial,  where  the  queue  size  is  110,  which  is  the  size  of  the  entire  data  set.  This  last 
trial  is  the  best-case  scenario  since  all  the  reports  are  sorted  according  to  their  Vol 
value.  Furthermore,  during  each  trial,  the  simulation  is  run  1 10  times.  During  each 
run  the  total  SA  gained  and  the  number  of  reports  analyzed  is  recorded. 

3.  Results 


The  experimental  results  were  collected  using  the  C3TRACE  default  reporting  and 
macro  features  to  tally  the  SA  gained  and  reports  analyzed.  The  results  were  plotted 
to  show  the  relationship  between  the  average  SA  gained  and  the  average  queue 
deficit. 

The  queue  deficit  is  the  difference  between  the  queue  size  used  for  the  trial  and  the 
average  number  of  reports  analyzed  across  all  the  runs  in  that  trial.  This  plot  shows 
that  for  the  Uniform  Data  Set,  the  SA  gained  does  not  improve  until  the  queue 
deficit  disappears  and  becomes  a  surplus  (Fig.  5).  In  other  words,  the  SA  begins  to 
improve  once  more  reports  are  sorted  than  the  simulation  is  able  to  analyze.  On  the 
contrary,  the  plot  showing  the  results  using  the  Normal  Data  Set  reveals  that  the  SA 
gained  never  improves  regardless  of  queue  deficit  (Fig.  6). 
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Uniform  Data  Set:  Situational  Awareness  Gained  vs 
Queue  Deficiency 


Queue  Size  minus  Reports  Anafyzed 

Fig.  5  Uniform  data  set  results 
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Normal  Data  Set:  Situational  Awareness  Gained  vs 
Queue  Deficiency 
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Fig.  6  Normal  data  set  results 


These  results  indicate  that  the  data  environment  has  a  role  in  determining  if  the  Vol 
metric  has  an  effect  on  SA  gained  in  this  simulation.  This  experiment  looked  at  2 
possible  data  distributions,  uniform  and  normal.  In  the  trials  using  the  uniform  data 
set,  the  Vol  metric  did  improve  the  SA  gained  once  the  queue  size  was  greater  than 
the  number  of  reports  analyzed.  If  the  queue  size  is  smaller  than  the  amount  of 
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reports  analyzed,  all  the  reports  in  the  queue  would  have  been  read  anyway,  and 
sorting  based  on  the  Vol  metric  gains  nothing.  To  illustrate,  if  the  queue  size  is  10, 
the  first  10  reports  in  the  shuffled  list  were  sorted  by  Vol  value  and  the  remaining 
100  were  left  in  their  shuffled  order.  If  the  simulation  analyzes  more  than  10 
reports,  the  sorting  was  meaningless  because  all  10  reports  were  read  in  the  end.  If 
the  simulation  only  reads  5  reports,  the  Vol  metric  improves  SA  gained  because  the 
best  5  reports  are  from  the  sorted  portion  of  the  list. 

On  the  contrary,  the  results  show  that  the  SA  gained  during  trials  with  the  normal 
data  set  do  not  change  under  any  circumstance  in  the  simulation.  Due  to  the  normal 
distribution,  approximately  68%  of  the  data  is  within  one  standard  deviation  of  the 
mean  Vol  value  (5  in  this  experiment).  Since  the  majority  of  the  reports  are  close 
to  the  same  Vol  value,  there  does  not  appear  to  be  enough  variation  to  make  sorting 
the  reports  by  Vol  value  meaningful. 

4.  Conclusion 


The  potential  to  assist  an  analyst  in  digesting  the  most  valuable  intelligence  first  is 
an  important  component  in  shortening  the  time  from  data  to  decisions.  In  this 
experiment,  the  simulation  results  showed  that  the  benefit  of  the  proposed  Vol 
metric  has  a  strong  dependence  on  the  nature  of  the  data  to  which  it  is  applied.  If 
you  consider  the  experimental  data  sets  to  be  the  endpoints  of  a  spectrum  of 
scenarios  (spread  out  versus  clustered  together),  the  results  show  that  the  impact  of 
the  Vol  metric  is  greatest  in  the  scenario  where  the  value  of  the  reports  are  evenly 
distributed.  It  is  ineffective  when  the  value  of  the  reports  are  clustered  closely 
together.  Further  research  is  necessary  to  gauge  the  impact  of  Vol  on  data  sets  that 
exist  between  the  endpoints  of  the  spectrum  and  to  attempt  to  discern  how  real- 
world  data  behave. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ARL 

US  Army  Research  Laboratory 

C3TRACE 

Command,  Control,  and  Communications  -  Techniques  for  Reliable 
Assessment  of  Concept  Execution 

HUMINT 

human  intelligence 

SA 

situational  awareness 

SME 

subject  matter  expert 

Vol 

value  of  information 
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